Demonstration at Evaluation 2024: Amplifying and Empowering Voices, annual conference of the American Evaluation Association, Portland, OR
Center for Statistical Training and Consulting, Michigan State University
2024-10-22
Let’s define some concepts.
… is achieved when investigators share all the materials required to exactly recreate the findings so that others can verify them or conduct alternative analyses.
Repeatable
Reproducible
Replicable
RR is a product of how we work, not which methods we use.
Several forces are promoting and enabling the push toward reproducibility.
Irreproducible < Reproducible < Replicated
Important
Reproducibility is an attainable minimum standard for science[1].
Pursuing reproducibility enacts our guiding principles[2]:
Data sharing and reproducibility initiatives
Understand the criteria, then apply principles, practices, & tools.
| Materials | Findings |
|---|---|
| Manuals & procedures | Statistics |
| Instruments & scoring rules | Coefficients & p-values |
| Codebooks | Confidence intervals |
| Methods applied | Effect sizes |
| Data mgt decisions | Model fit indices |
| Data files | Figures |
| Software & analysis scripts | Tables |
“Captain, you’re asking me to work with equipment which is hardly very far ahead of stone knives and bearskins.”
Star Trek (1966) - S01E28 The City on the Edge of Forever
A compendium organizes digital files so others can review or use them to reproduce results, or do new analyses.[18] It should:
… are folders of digital files designed for sharing code & help documentation.[19] They
Tip
Use an R package for a research compendium![18]
… are folders of files being tracked by Git[17] for version control purposes.[20,21] They:
Tip
Put your research compendium in a Git repository!
… are folders of files that:
*.Rproj)Tip
Turn your compendium into an RStudio project!
… are folders of files that allow:
Tip
Turn a subfolder of your compendium into a Quarto project!
MyStudy/ [Compendium, Git repository, R package, RStudio project]
- .git/ [Hidden folder, holds Git tracking database]
- .Rproj.user/ [Hidden folder, holds RStudio temporary files]
- data/ [Holds R data files created by scripts]
- man/ [Holds R help files for package & custom functions]
- R/ [Holds R scripts w/ custom functions]
- scripts/ [Quarto project, holds dynamic documents]
- extdata/ [Holds external data files to be imported]
- output/ [Holds rendered output]
- .gitignore [Tells Git what to omit from tracking]
- DESCRIPTION [R package meta-data]
- MyProject.Rproj [RStudio project file & settings]
- NEWS.md [News for users re: changes to package]
- README.Rmd [Dynamic document, creates README.md]
- README.md [Rendered output, R package documentation]
What should you preserve?
Let’s take a look at actual dynamic documents!